System and method for manipulating deformable objects

ABSTRACT

A method for manipulating a deformable object includes determining respective 3D position of one or more markers on the deformable object held by a robotic manipulator; determining a deformation model of the deformable object by mapping movement of the robotic manipulator and movement of one or more markers; and controlling the robotic manipulator based on the determined deformation model to manipulate the deformable object so as to move the one or more markers into respective target position.

TECHNICAL FIELD

The invention relates to system and method for manipulating deformableobjects, and particularly, although not exclusively, to system andmethod for manipulating fabrics.

BACKGROUND

Robots are widely used in the manufacturing industry to free humanworkers from tedious repetitive tasks. Techniques for rigid bodymanipulation have been developed and applied in industry for decades.However, to date, there are few deformable object manipulation methodsthat are mature enough for practical applications.

Compared to the rigid object manipulation, tasks related to deformableobject manipulation have many additional challenges. One example ofdeformable object manipulation is the assembly of cloth pieces tofixtures. In this task, a piece of cloth with holes has to be alignedwith a fixture made by a set of vertical locating pins. The assembledcloth pieces are then sent to the sewing machine for sewing operations.The task can be efficiently performed by a human worker without anytraining, but may be difficult for a robot. For instance, the wrinklesgenerated during operation will interfere with feature extraction andtracking procedures that are critical for perception feedbacks. Also,the highly deformable property of the cloth will lead to unpredictablechanges in the size and shape of the holes during the manipulation.These challenges make it difficult to achieve an accurate and reliablerobotic manipulation control for clothes.

SUMMARY OF THE INVENTION

It is an object of the invention to address the above needs, to overcomeor substantially ameliorate the above disadvantages or, more generally,to provide an improved system and method for manipulating deformableobjects.

In accordance with a first aspect of the invention, there is provided amethod for manipulating a deformable object, comprising: determiningrespective 3D position of one or more markers on the deformable objectheld by a robotic manipulator; determining a deformation model of thedeformable object by mapping movement of the robotic manipulator andmovement of one or more markers; and controlling the robotic manipulatorbased on the determined deformation model to manipulate the deformableobject so as to move the one or more markers into respective targetposition.

In one embodiment of the first aspect, the determination of respective3D position of one or more markers comprises: detecting the one or moremarkers on the deformable object;

tracking the one or more markers on the deformable object as thedeformable object is manipulated by the robotic manipulator; andcalculating respective 3D position of the one or more markers based onthe detection and tracking result.

In one embodiment of the first aspect, the detection of the one or moremarkers comprises: detecting, using a sensor, respective 2D position ofthe one or more markers. Preferably, the detection uses a templatematching method that includes: comparing data obtained by the sensorwith reference data for the one or more markers to determine similarity;and determining the presence of the one or more markers based onsimilarity result. The template matching method may further include:binarizing the similarity result based on a threshold and applying amorphological operation to the binarized result to facilitatedetermination of the presence of the one or more markers.

In one embodiment of the first aspect, the tracking of the one or moremarkers comprises: detecting, using a sensor, respective 2D position ofthe one or more markers as the deformable object is manipulated.Preferably, the tracking uses a kernelized correlation filters trackingmethod that includes: comparing data obtained by the sensor at differentinstances using a classifier. The classifier can be trained using dataobtained by the sensor at different instances.

In one embodiment of the first aspect, the deformation model isdetermined based on online learning.

In one embodiment of the first aspect, the mapping is based on aGaussian mapping function.

In one embodiment of the first aspect, the deformation model iscontinuously determined as the robotic manipulated is controlled basedon the determined deformation model.

In one embodiment of the first aspect, controlling the roboticmanipulator to manipulate the deformable object includes: moving thedeformable object to align the one or more markers with the respectivetarget position; and moving the deformable object to bring the one ormore markers to the respective target position. In one example, movingthe deformable object to bring the one or more markers to the respectivetarget position comprises moving the deformable object along a singledirection, for example a vertical direction.

In one embodiment of the first aspect, the sensor comprises one or moredepth cameras. The depth cameras may be one or more of ranging camera,flash LIDAR, time-of-flight camera, RGBD camera, etc. Preferably, thesensor comprises an RGBD camera.

In one embodiment of the first aspect, the robotic manipulator comprisesat least two robotic arms each having an end-effector for holding thedeformable object. Preferably, the end-effector includes one or moregripping plates, each of which includes a gripping surface with one ormore nubs.

In one embodiment of the first aspect, the deformable object is a pieceof fabric, for example, cloth. The one or more markers may be holes inthe deformable object. The holes may be of different shape, e.g.,circular, squared, oval, oblong, etc.

In accordance with a second aspect of the invention, there is provided asystem for manipulating a deformable object, comprising a controllerarranged to: determine respective 3D position of one or more markers onthe deformable object held by a robotic manipulator; determine adeformation model of the deformable object by mapping movement of therobotic manipulator and movement of one or more markers; and control therobotic manipulator based on the determined deformation model tomanipulate the deformable object so as to move the one or more markersinto respective target position.

In one embodiment of the second aspect, the system further comprises asensor, operably connected with the controller, for detecting andtracking the one or more markers on the deformable object.

In one embodiment of the second aspect, the sensor comprises one or moredepth cameras. The depth cameras may be one or more of ranging camera,flash LIDAR, time-of-flight camera, RGBD camera, etc. Preferably, thesensor comprises an RGBD camera.

In one embodiment of the second aspect, the system further comprises therobotic manipulator operably connected with the controller. Preferably,the robotic manipulator comprises at least two robotic arms each havingan end-effector for holding the deformable object. The end-effector mayinclude one or more gripping plates each having a gripping surface withone or more nubs.

In one embodiment of the second aspect, the controller is arranged toprocess data obtained by the sensor, to calculate respective 3D positionof the one or more markers.

In one embodiment of the second aspect, the controller is arranged to:compare data obtained by the sensor with reference data for the one ormore markers to determine similarity; binarizing the similarity resultbased on a threshold and applying a morphological operation to thebinarized result; and accordingly, determine the presence of the one ormore markers.

In one embodiment of the second aspect, the controller is arranged to:compare data obtained by the sensor at different instances using aclassifier, wherein the classifier is trained using data obtained by thesensor at different instances.

In one embodiment of the second aspect, the deformation model isdetermined based on online learning and the mapping is based on aGaussian mapping function.

In accordance with a third aspect of the invention, there is provided anon-transitory computer readable medium for storing computerinstructions that, when executed by one or more processors, causes theone or more processors to perform a method for manipulating a deformableobject, comprising: determining respective 3D position of one or moremarkers on the deformable object held by a robotic manipulator;determining a deformation model of the deformable object by mappingmovement of the robotic manipulator and movement of one or more markers;and controlling the robotic manipulator based on the determineddeformation model to manipulate the deformable object so as to move theone or more markers into respective target position.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example,with reference to the accompanying drawings in which:

FIG. 1 is a schematic diagram illustrating the relationship betweenholes in the deformable object and movement of the robotic manipulatorin accordance with one embodiment of the invention;

FIG. 2 is a flow diagram illustrating a method for manipulating adeformable object in one embodiment of the invention;

FIGS. 3A-3F illustrate different metrics used in template matching inthe method of FIG. 2, wherein the upper image shows the positions ofcandidates on the original RGB images and the lower image shows thepositions of candidates on the response image where the intensityrepresents the response level, and each white box represents a validmatching candidate, in particular: FIG. 3A represents squareddifference; FIG. 3B represents normalized squared difference; FIG. 3Crepresents cross correlation; FIG. 3D represents normalized crosscorrelation; FIG. 3E represents correlation coefficient; and FIG. 3Fshows normalized correlation coefficient;

FIG. 4 illustrates the images (upper row) and the depth image (lowerrow) obtained using the tracking algorithm in one embodiment of theinvention at different time points;

FIGS. 5A and 5B illustrate performance of our perception system, and inparticular, FIG. 5A shows the original depth image when self-occlusionhappens; and FIG. 5B shows the detection of holes from the noisy data,as shown by the red contours, using the method in one embodiment of theinvention;

FIG. 6 is a picture showing the set-up of a system for manipulating adeformable object in one embodiment of the invention;

FIG. 7 is a picture showing an assistive component attached to theend-effector of the robotic manipulator for reducing the amount and sizeof wrinkles and folds during the manipulation process in one embodimentof the invention;

FIG. 8 is a picture showing items (from left to right: the stretchablecloth piece, the fixture, and the un-stretchable cloth piece) used inExperiment 1 performed using the system of FIG. 6;

FIG. 9A is a picture showing initial state for the assembly process inExperiment 1 when the stretchable cloth piece is used;

FIG. 9B is a picture showing final state for the assembly process inExperiment 1 when the stretchable cloth piece is used;

FIG. 9C is a picture showing initial state for the assembly process inExperiment 1 when the unstretchable cloth piece is used;

FIG. 9D is a picture showing final state for the assembly process inExperiment 1 when the unstretchable cloth piece is used;

FIG. 10A is a picture showing items (from left to right: the fixture andthe un-stretchable cloth piece) used in Experiment 2 performed using thesystem of FIG. 6;

FIG. 10B is a schematic illustrating the configuration of holes in thecloth piece in FIG. 10A;

FIG. 10C is a schematic illustrating the configuration of pins on thefixture in FIG. 10A;

FIG. 11A is a picture showing initial state for the assembly process inExperiment 2;

FIG. 11B is a picture showing final state for the assembly process inExperiment 2;

FIG. 12A is a graph showing measured errors between the currentpositions of holes and target positions during manipulation of thestretchable material in Experiment 1;

FIG. 12B is a graph showing measured errors between the currentpositions of holes and target positions during manipulation of theunstretchable material in Experiment 1;

FIG. 13 is a graph showing measured errors between the current positionsof holes and target positions during manipulation of the material inExperiment 2; and

FIG. 14 is a functional block diagram of an information handling systemthat can be used to implement the method of FIG. 2.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 6 shows a system 600 for manipulating a deformable object in oneembodiment of the invention. The system 600 includes a controller 602arranged to: determine respective 3D position of one or more markers onthe deformable object held by a robotic manipulator 606; determine adeformation model of the deformable object by mapping movement of therobotic manipulator and movement of one or more markers; and control therobotic manipulator based on the determined deformation model tomanipulate the deformable object so as to move the one or more markersinto respective target position. In one example the deformable objectmay be a piece of cloth, and the markers may be one or more holes in thecloth.

The system 600 also includes a sensor 604, operably connected with thecontroller, for detecting and tracking the one or more markers on thedeformable object. The sensor 604 includes one or more depth cameras,which is preferably RGBD camera but can also be may be ranging camera,flash LIDAR, time-of-flight camera, etc.

The robotic manipulator 606 is operably connected with the controller602. The robotic manipulator 606 can have 6 degrees of freedom, and itincludes at least two robotic arms each having an end-effector forholding the deformable object. The end-effector may include one or moregripping plates 712 each having a gripping surface with one or more nubs714, as shown in FIG. 7.

In one embodiment, the controller 602 is arranged to process dataobtained by the sensor, to calculate respective 3D position of the oneor more markers. The controller 602 is further arranged to: compare dataobtained by the sensor 604 with reference data for the one or moremarkers to determine similarity; binarizing the similarity result basedon a threshold and applying a morphological operation to the binarizedresult; and accordingly, determine the presence of the one or moremarkers. The controller 602 may compare data obtained by the sensor 604at different instances using a classifier, wherein the classifier istrained using data obtained by the sensor at different instances. In apreferred embodiment, the deformation model is determined based ononline learning and the mapping is based on a Gaussian mapping function.

In one embodiment of the invention, the system 600 can be applied to acloth assembly task, and in particular, a manipulation procedure wherethe robotic end-effectors manipulate the cloth pieces in a way such thatthe holes in the cloth pieces will be aligned with the set of pins on afixture and eventually the pins will go through the holes to accomplishthe assembly. This task can be divided into two subtasks: to retrievethe 3D position of each hole, and to control the robot arms to move thecloth pieces into the targeted position based on the positions of holesand pins.

In this manipulation task, a function F(⋅) is used to describe therelationship between the relative displacement of current positions ofholes c and the velocity of the manipulated points x on the object whichis grabbed by the end-effectors

δx=F(δc).  (1)

FIG. 1 illustrates the relationship between markers in the form of holesin the deformable object and movement of the robotic manipulator(grippers) in accordance with one embodiment of the invention. Given thetarget position of holes c^(d) and current position of the holes c_(*),the required manipulation velocity {dot over (x)} for the end-effectorsin each control cycle can be obtained using

{dot over (x)}=F(η·(c _(d) −c _(*))),  (2)

where η is an appropriate feedback gain to be tuned for fastconvergence; c^(d) and c_(*) are the target and current positions of theholes in the 3D workspace, respectively. In the present embodiment, thefunction F(⋅) will change according to the shape and material of thecloth piece as well as the patterns of holes in the cloth piece.

The system 600 is adapted for the general deformable object control. Therobotic manipulator 606 may have a sufficient number of degrees offreedom for dexterous manipulation of cloth pieces. The roboticmanipulator 606 supports velocity control, i.e., controlled by thecontroller 602, to implement the feedback control policy in Equation 2.At least one sensor, preferably RGBD sensor 604, is arranged in thesystem 600 to provide 3D measurements for the deformable object.

The assembly task can be divided into two sub-tasks, and the system 600of the present embodiment serves two major functions: perception andcontrol. Perception relates to the detection and tracking of the holesin the cloth piece, and computation of the 3D position of each hole.Control refers to the generation of appropriate commands to drive therobotic arm based on the current positions of holes and pins.

FIG. 2 illustrates a method 200 for manipulating a deformable object inone embodiment of the invention. The method generally involves,determining respective 3D position of one or more markers on thedeformable object held by a robotic manipulator; determining adeformation model of the deformable object by mapping movement of therobotic manipulator and movement of one or more markers; and controllingthe robotic manipulator based on the determined deformation model tomanipulate the deformable object so as to move the one or more markersinto respective target position. Determination of respective 3D positionof one or more markers comprises: detecting the one or more markers onthe deformable object; tracking the one or more markers on thedeformable object as the deformable object is manipulated by the roboticmanipulator; and calculating respective 3D position of the one or moremarkers based on the detection and tracking result. Detail of the method200 is provided in more detail below.

Perception

The perception sub-task includes two steps: recognizing and trackingholes using the RGB-D data stream, and calculating the positions ofholes based on the depth information.

In this embodiment, for performing recognition, a template matchingmethod to determine the 2D positions of holes in the first frame. Inparticular, the difference between a template patch about the hole andall possible regions in the given image is first computed to find a setof candidate regions that have the highest similarity with the template.The regions with local maximum similarity as the hole positions are thenselected.

To improve the hole recognition performance, in this embodiment, thematching similarity result is first binarized using a fixed thresholdand then morphological operations including dilation and erosion areapplied to the binarized result to further separate potential connectedregions. In this way, the point with local maximum similarity insideeach region can represent a valid matching target for a hole.

Various schemes can be used to calculate the similarity between tworegions. FIG. 3A-3F show the comparison of different matching metrics,concluding squared difference (FIG. 3A), normalized squared difference(FIG. 3B), cross correlation (FIG. 3C), normalized cross correlation(FIG. 3D), correlation coefficient (FIG. 3E), and normalized correlationcoefficient (FIG. 3F). In the present embodiment, the one can besttolerate the errors caused by the deformation and thus provide accuratematching results is chosen. According to the comparison results shown inFIG. 3A-3F, the normalized correlation coefficient metric for thetemplate matching is chosen, which is computed as

$\begin{matrix}{{{R\left( {x,y} \right)} = \frac{\sum\limits_{x^{\prime},y^{\prime}}\; \left( {{T\left( {x^{\prime},y^{\prime}} \right)} \cdot {I\left( {{x + x^{\prime}},{y + y^{\prime}}} \right)}} \right)}{\sqrt{\sum\limits_{x^{\prime},y^{\prime}}\; {{T\left( {x^{\prime},y^{\prime}} \right)}^{2} \cdot {\sum\limits_{x^{\prime},y^{\prime}}\; {I\left( {{x + x^{\prime}},{y + y^{\prime}}} \right)}^{2}}}}}},} & (3)\end{matrix}$

where R(x; y) is the similarity response at the location (x; y) in thecurrent image I, and T is the template patch.

As the size and the shape of a hole may change during the manipulation,a robust and time-efficient tracker for the holes is necessary. In thisembodiment, the Kernelized Correlation Filters (KCF) tracking algorithmprovided in J. F. Henriques, R. Caseiro, P. Martins, and J. Batista,“High-speed tracking with kernelized correlation filters,” IEEETransactions on Pattern Analysis and Machine Intelligence, vol. 37, no.3, pp. 583-596, 2015 is used because the KCF algorithm uses circulantmatrices with Fourier transform to reduce the computational cost andleverages kernelized regression to guarantee the precision.

The KCF tracking algorithm trains a classifier using the patchesextracted near the current positions of holes and uses the classifier toselect the candidate hole regions with the highest response in the nextframe. For each valid candidate, a square with constant size isgenerated to bound the hole. The regions inside squares are consideredas the region of interest in the current frame for further processingand will be fed to the KCF tracker algorithm as new targets. Inparticular, the initial tracking targets are the valid matching regionsfrom the template matching method.

This tracking algorithm can monitor many holes simultaneously and keepstable tracking even when the shape of the cloth piece has been changedsignificantly during the manipulation, as demonstrated in FIG. 4.

To calculate the positions of holes, in the present embodiment, thegeometric center of holes is used to describe the positions of holes inthe cloth piece. Given {p_(i)}ϵ

³ for 1≤i≤N which denote the position of points on the boundary of ahole, the geometric center C of the hole can be computed as

C=(p ₁ +p ₂ + . . . +p _(N))/N  (4)

To accomplish the above measurement, 2D points on the hole boundary isfirst extracted from the RGB image. These 2D points correspond to theprojection of the hole onto the image plane, and thus they should form aclosed contour in 2D. The searching area for these contours shrinks fromthe whole image into the areas of interest provided by KCF trackingalgorithm, because holes may only exist in these areas. In ideal caseswithout occlusion, wrinkles or illumination change, the steepestintensity change in the region of interest should be perpendicular tothe hole boundary. Thus, after applying the Canny edge detector, thereshould be only one closed contour inside the region of interest, whichcorresponds to the hole boundary. Due to the one-to-one correspondencebetween the 2D and depth images, the depth values of the points on thehole boundary can be easily computed. After taking the intrinsicparameters of the camera into account, the 3D positions of the holeboundary points can be computed.

Due to calibration error of the RGBD camera and noises caused by lightreflection, not all points on the contour can have a valid depth value.To address this issue, in the present embodiment, the contour isenlarged evenly around its 2D center until all its boundary pixels havevalid depth values. In particular, a pixel u_(i)ϵ

² on the hole boundary will be transformed into the 2D position

$\begin{matrix}{{u_{i}^{\prime} = {{\alpha \times u_{i}} + {\left( {1 - \alpha} \right){\sum\limits_{i}\; u_{j}}}}},} & (5)\end{matrix}$

where α is the parameter for the enlarging ratio.

FIGS. 5A and 5B show that the system in the present embodiment candetect closed contours that provide a high-quality approximation to theholes' 3D positions, even when the depth image is noisy or has lost someinformation due to self-occlusion.

Control In our approach, the complete control pipeline can be dividedinto two stages: stage I—the robot moves the cloth piece to the pins;stage II—the robot controls the end-effectors to put down cloth piecesvertically.

In operation, the function between the relative displacement of theposition of holes and the velocity of manipulated points depends on thecloth piece to be manipulated, because the function varies with thematerial, shape, or hole pattern of the cloth piece. In the presentembodiment, stage I applies a general approach using an online learningalgorithm to model this function adaptively during the manipulationprocess.

According to δx=F(δc), the present embodiment requires a model to learnthe mapping F(⋅) which predicts the required control velocity δx givenholes' velocity δc. Due to the highly nonlinear property of the functionF(⋅), in the present embodiment, the mapping function F is formulated asa Gaussian process:

F˜GP(0,k(δc,δc′)),  (6)

where

${k\left( {{\delta \; c},{\delta \; c^{\prime}}} \right)} = {\exp \left( \frac{- {{{\delta \; c} - {\delta \; c^{\prime}}}}^{2}}{2\sigma_{RBF}^{2}} \right)}$

is the covariance function called radial basis function (RBF) andparameter σRBF sets the spread of the kernel. Accordingly,

$\begin{matrix}{\begin{bmatrix}{\delta \; X} \\\overset{.}{x}\end{bmatrix} \sim {\mspace{11mu} \left( {0,\begin{bmatrix}{{K\left( {{\delta \; C},{\delta \; C}} \right)} + {\sigma_{n}^{2}I}} & {K\left( {{\delta \; C},e} \right)} \\{K\left( {e,{\delta \; C}} \right)} & {K\left( {e,e} \right)}\end{bmatrix}} \right)}} & (7)\end{matrix}$

where e=η·(c^(d)−c_(*)) is the difference between target positions c^(d)and the current positions of holes c_(*); δC={δc_(t)}_(i=1) ^(N) andδX={δx_(t)}_(t=1) ^(N) are the stack of the hole displacements andend-effector velocities collected during the manipulation; and σ_(n) isthe noise of the data measurements.

The Gaussian process regression in the present example will experiencetwo phases during the manipulation process: the initialization phase andthe manipulation phase (or online learning phase).

During the initialization phase, the controller gathers batches of data(δX, δC) to fit the GP model, which is also the training process of themodel. The GPR controller receives the velocity of holes and manipulator(gripper) as input; however, when the and manipulator is not moving,i.e., when the velocity of manipulated points is zero, the controllershould not take this entry as a valid sample since it does not provideany information about the deformable model of the cloth piece. Thus, thecontroller will only keep the data pairs where the velocity of themanipulator is non-zero.

In the manipulation phase, the GPR will predict the required speed {dotover (x)} of each gripper to drive the cloth piece toward the goalstate, where

{dot over (x)}=K ^(T)(δC,δe)·[K(δC,δC)+σ_(n) ² I]⁻¹ ·δX.  (8)

Once the GP-based movement command is executed, the cloth will moveimmediately and a new pair of data is generated where δx_(*) is thespeed of end-effectors and δx_(*) is the new visual feedback obtainedfrom the perception. That is, the number of data samples for GPR willkeep increasing during the manipulation. These new data samples canupdate the GP model about the possibly changing deformation parametersof the object. Since the GP model is keep updating during manipulation,this phase can be regarded as the online learning phase of GPR.

As the storage and computation cost of the covariance matrix K in GPgrows quadratically, a maximum size of the covariance matrices must beset to enable the controller to operate at a high frequency. Thus, anupper limit N is set to restrict total stored samples and thus the upperlimit size of covariance matrices is N². If a new sample is obtainedwhile there are already N samples, a replacement policy is adopted tomaintain the total amount of samples unchanged. In particular, anearlier sample with the largest kernelized similarity with other sampleswill be replaced by the new sample. This strategy helps the model tokeep data items that contribute to the model simulation, and exposes theGPR model to more deformation variations. In addition, the controllerwill always use the most recent data to determine the movement in thenext step, and thus can safely adapt to the changing deformation model.The entire replacement procedure can be accomplished efficiently with

(N²) complexity.

After stage I, the cloth piece has been placed in a suitable positionwhere the holes are aligned above the corresponding pins. In stage II,the controller only needs to enable the end effectors to move along asingle direction, the direction of the pins, with the same speed toavoid any possible undesirable deformations. In this way, the clothpiece will be readily fixed onto the fixture.

Experiments

A set of experiments were conducted to assess the system and method ofthe present embodiment. The experiment is performed using a system,illustrated in FIG. 6, built based on a Yumi 14400 dual-armcollaborative robot with two RGBD cameras (Intel Realsense SR300 andZR300) mounted. In this set-up, the two cameras have differentperception fields and resolutions and are used in different scenarioscategorized by the size of the workspace and the cloth piece. OpenCV isused or image processing and hole tracking. The controller sendsvelocity control commands to the robot and recalculates necessarytrajectory variation automatically based on perception feedbacks.

Two different types of cloth pieces were used in experiments, one madeof stretchable material and the other made of unstretchable material.The diameter of each hole in a cloth piece is about 1 cm, and thediameters of pins are around 0.7 cm. The distances between two holes andthe corresponding pins may not be equal, in order to fulfill certaingarment design requirements which needs to stretch or loose the clothpiece. The error tolerance for the distance between the centroidposition of the hole and the corresponding target point is 0.005 m.

Unlike other cloth manipulation tasks such as sewing or folding that canbe done by laying the cloth piece flat on a surface, the assembly tasksin the present example includes stretching and shrinking operationswhich require the cloth piece to be lifted. Because the inner part ofthe cloth is floating, the cloth piece is likely to have undesiredwrinkles and pleats during the manipulation, which may complicate thetracking process. To alleviate this issue, an assistive component asshown in FIG. 7 is attached to the end-effectors to reduce the amountand size of folds. AS shown in FIG. 7, the assistive component is agripping plate 712, a gripping surface of which has multiple nubs 714.

For control, the maximum velocity of the end-effectors is set as 0.04m/s and the control frequency as 30 Hz. The initialization of GPR modelis accomplished by movements either using pre-programmed tracks or underhuman instructions for the purpose of gathering enough data to describethe deformation model. The kernel spread width RR Fin the covariancecomputation is set as 0.6, the noise level is set as 0.001, and themaximum size limit of the observation matrix N is 300.

For simplicity, the processes of controlling the robot to grasp thecloth and keep the location of fixtures unchanged in the aboveexperiments are omitted.

Two experiments were performed to test the performance of our clothassembly system.

Experiment 1 is designed to test the adaptability of the system of thepresent embodiment to materials with different deformation properties.In this experiment, a fixture and two different types of cloth pieceseach with two holes as shown in FIG. 8 are used. The distance betweencenters of two holes in the stretchable cloth piece is 6 cm and thedistance between centers of two holes in the unstretchable cloth pieceis 7 cm (when the cloth is flattened on a smooth plane withoutsubjecting to any external forces). The distance between pins of thefixture is 6.3 cm, in order to test the stability of thisvisual-servoing system under situations where extruding the stretchablematerial or shrinking the unstretchable material is demanded. Thedimension of relative displacement vector is 6 for both the positions cof two holes and the positions x of manipulator or grippers.

FIGS. 9A and 9B show the initial state and the final state for theassembly process in Experiment 1 when the stretchable cloth piece isused; while FIGS. 9C and 9D show the initial state and the final statefor the assembly process in Experiment 1 when the unstretchable clothpiece is used. The curves for the measured errors between the currentposition of holes and target positions during manipulation in thisexperiment are shown in FIGS. 12A and 12B, where it can be observed thatthe errors drop quickly to zero for both materials. From these results,it can be concluded that the visual-servoing system in the presentembodiment can handle materials with different stiffness well and canaccomplish tasks properly when different manipulation skills likestretching and squeezing are required.

Experiment 2 is designed to test the performance of the system of thepresent embodiment when it is used to handle tasks where the dimensionof target position is larger than the degrees-of-freedom of the grippers(a difficulty from the control perspective). In this experiment, a clothpiece with four holes and a fixture as shown in FIG. 10A are used. Thehole arrangement of the cloth piece is shown in FIG. 10B and the pinarrangement of the fixture is shown in FIG. 10C. The pin arrangement andthe hole arrangement are of slightly different geometry. Since there arefour holes in total, the dimension of the vector c corresponding to thepositions of holes is 12 while the dimension of the vector xcorresponding to the end-effector position is 6.

FIGS. 11A and 11B show the initial state and the final state for theassembly process in Experiment 2. The curve for the errors between thecurrent positions of holes and target positions during manipulation isshown in FIG. 13. It can be observed that the error decreases to zerowithin acceptable time. This experiment validates the system of thepresent embodiment, showing that it can handle tasks in which the targetrequirement has higher dimensions than the degrees-of-freedom of themanipulator.

Referring to FIG. 14, there is shown a schematic diagram of an exemplaryinformation handling system 1400 that can be used to implement themethod of FIG. 2 in one embodiment of the invention. Preferably, theinformation handling system 1400 may have different configurations, andit generally comprises suitable components necessary to receive, storeand execute appropriate computer instructions or codes. The maincomponents of the information handling system 1400 are a processing unit1402 and a memory unit 1404. The processing unit 1402 is a processorsuch as a CPU, an MCU, etc. The memory unit 1404 may include a volatilememory unit (such as RAM), a non-volatile unit (such as ROM, EPROM,EEPROM and flash memory) or both. Preferably, the information handlingsystem 1400 further includes one or more input devices 1406 such as akeyboard, a mouse, a stylus, a microphone, a tactile input device (e.g.,touch sensitive screen) and a video input device (e.g., camera). Theinformation handling system 1400 may further include one or more outputdevices 1408 such as one or more displays, speakers, disk drives, andprinters. The displays may be a liquid crystal display, a light emittingdisplay or any other suitable display that may or may not be touchsensitive. The information handling system 1400 may further include oneor more disk drives 1412 which may encompass solid state drives, harddisk drives, optical drives and/or magnetic tape drives. A suitableoperating system may be installed in the information handling system1400, e.g., on the disk drive 1412 or in the memory unit 1404 of theinformation handling system 1400. The memory unit 1404 and the diskdrive 1412 may be operated by the processing unit 1402. The informationhandling system 1400 also preferably includes a communication module1410 for establishing one or more communication links (not shown) withone or more other computing devices such as a server, personalcomputers, terminals, wireless or handheld computing devices. Thecommunication module 1410 may be a modem, a Network Interface Card(NIC), an integrated network interface, a radio frequency transceiver,an optical port, an infrared port, a USB connection, or otherinterfaces. The communication links may be wired or wireless forcommunicating commands, instructions, information and/or data.Preferably, the processing unit 1402, the memory unit 1404, andoptionally the input devices 1406, the output devices 1408, thecommunication module 1410 and the disk drives 1412 are connected witheach other through a bus, a Peripheral Component Interconnect (PCI) suchas PCI Express, a Universal Serial Bus (USB), and/or an optical busstructure. In one embodiment, some of these components may be connectedthrough a network such as the Internet or a cloud computing network. Aperson skilled in the art would appreciate that the information handlingsystem 1400 shown in FIG. 14 is merely exemplary, and that differentinformation handling systems 1400 may have different configurations andstill be applicable in the invention.

Although not required, the embodiments described with reference to theFigures can be implemented as an application programming interface (API)or as a series of libraries for use by a developer or can be includedwithin another software application, such as a terminal or personalcomputer operating system or a portable computing device operatingsystem. Generally, as program modules include routines, programs,objects, components and data files assisting in the performance ofparticular functions, the skilled person will understand that thefunctionality of the software application may be distributed across anumber of routines, objects or components to achieve the samefunctionality desired herein.

It will also be appreciated that where the methods and systems of theinvention are either wholly implemented by computing system or partlyimplemented by computing systems then any appropriate computing systemarchitecture may be utilized. This will include stand-alone computers,network computers and dedicated hardware devices. Where the terms“computing system” and “computing device” are used, these terms areintended to cover any appropriate arrangement of computer hardwarecapable of implementing the function described.

The above embodiments of the invention provide a system for manipulatingdeformable objects, e.g., for assembling cloth pieces. The system canadapt to the variation in size, shape, and material of the cloth pieceand is robust with respect to the pattern of holes in the cloth piece.In particular, an online learning algorithm is used to learn acontroller, which has the changing speed of holes in the cloth piece asthe input and the moving velocity of the end-effectors as the output.The controller will be continuously updated with the new sensormeasurements in order to maintain the latest knowledge about therelation between the deformation and the movement of the end-effectors.The method of the invention requires no prior knowledge about thedeformation parameters of cloth pieces and thus is applicable forgeneral assembly tasks involving deformable objects. The aboveembodiments of the invention also provide a method for manipulatingdeformable objects. The method in the above embodiments is applicable togeneral fabrics by using the model-free framework. Such model can learnthe object's deformation parameters online, leading to the tolerance tounknown or uncertain deformation parameters. A Gaussian Process (GP)model is used in one embodiment to better encode the nonlinearity in thedeformation behaviour of the object. The system and methods in the aboveembodiments are applicable solution to the cloth assembly problem andcan make an important component of the entire pipeline of automatedgarment manufacturing.

It will be appreciated by persons skilled in the art that numerousvariations and/or modifications may be made to the invention as shown inthe specific embodiments without departing from the spirit or scope ofthe invention as broadly described. For example, the system and methodcan be applied to manipulate different deformable objects such as cloth,paper, garments, etc. The markers on the deformable objects need not beholes, but can be, for example, identification marks. The number ofmarkers can be freely chosen. The system and method can be equallyapplied to manipulate rigid objects that are substantially notdeformable during manipulation. The system and method can be integratedas a robotic sewing system and method. The robotic manipulator couldtake any other form, with two or more robotic arms. The sensor can beany depth camera operable to provide 2D image and depth image. Thecontroller may be implemented using one or more computing devices, CPU,MCU, logic components, etc. The present embodiments are, therefore, tobe considered in all respects as illustrative and not restrictive.

1. A method for manipulating a deformable object, comprising:determining respective 3D position of one or more markers on thedeformable object held by a robotic manipulator; determining adeformation model of the deformable object by mapping movement of therobotic manipulator and movement of one or more markers; and controllingthe robotic manipulator based on the determined deformation model tomanipulate the deformable object so as to move the one or more markersinto respective target position.
 2. The method of claim 1, wherein thedetermination of respective 3D position of one or more markerscomprises: detecting the one or more markers on the deformable object;tracking the one or more markers on the deformable object as thedeformable object is manipulated by the robotic manipulator; andcalculating respective 3D position of the one or more markers based onthe detection and tracking result.
 3. The method of claim 2, wherein thedetection of the one or more markers comprises: detecting, using asensor, respective 2D position of the one or more markers.
 4. The methodof claim 3, wherein the detection uses a template matching method thatincludes: comparing data obtained by the sensor with reference data forthe one or more markers to determine similarity; and determiningpresence of the one or more markers based on similarity result.
 5. Themethod of claim 4, wherein the template matching method furtherincludes: binarizing the similarity result based on a threshold andapplying a morphological operation to the binarized result to facilitatedetermination of the presence of the one or more markers.
 6. The methodof claim 2, wherein the tracking of the one or more markers comprises:detecting, using a sensor, respective 2D position of the one or moremarkers as the deformable object is manipulated.
 7. The method of claim6, wherein the tracking uses a kernelized correlation filters trackingmethod that includes: comparing data obtained by the sensor at differentinstances using a classifier.
 8. The method of claim 7, wherein theclassifier is trained using data obtained by the sensor at differentinstances.
 9. The method of claim 1, wherein the deformation model isdetermined based on online learning.
 10. The method of claim 1, whereinthe mapping is based on a Gaussian mapping function
 11. The method ofclaim 1, wherein the deformation model is continuously determined as therobotic manipulated is controlled based on the determined deformationmodel.
 12. The method of claim 1, wherein controlling the roboticmanipulator to manipulate the deformable object includes: moving thedeformable object to align the one or more markers with the respectivetarget position; and moving the deformable object to bring the one ormore markers to the respective target position.
 13. The method of claim12, wherein moving the deformable object to bring the one or moremarkers to the respective target position comprises moving thedeformable object along a single direction.
 14. The method of claim 3,wherein the sensor comprises one or more depth cameras.
 15. The methodof claim 6, wherein the sensor comprises one or more depth cameras. 16.The method of claim 14, wherein the sensor comprises an RGBD camera. 17.The method of claim 1, wherein the robotic manipulator comprises atleast two robotic arms each having an end-effector for holding thedeformable object.
 18. The method of claim 17, wherein the end-effectorincludes one or more gripping plates, each gripping plate has a grippingsurface with one or more nubs.
 19. The method of claim 1, wherein thedeformable object is a piece of fabric.
 20. The method of claim 1,wherein the one or more markers are holes in the deformable object. 21.A system for manipulating a deformable object, comprising a controllerarranged to: determine respective 3D position of one or more markers onthe deformable object held by a robotic manipulator; determine adeformation model of the deformable object by mapping movement of therobotic manipulator and movement of one or more markers; and control therobotic manipulator based on the determined deformation model tomanipulate the deformable object so as to move the one or more markersinto respective target position.
 22. The system of claim 21, furthercomprising a sensor, operably connected with the controller, fordetecting and tracking the one or more markers on the deformable object.23. The system of claim 22, wherein the sensor comprises one or moredepth cameras.
 24. The system of claim 22, wherein the sensor comprisesan RGBD camera.
 25. The system of claim 21, further comprising therobotic manipulator operably connected with the controller, wherein therobotic manipulator comprises at least two robotic arms each having anend-effector for holding the deformable object.
 26. The system of claim25, wherein the end-effector includes one or more gripping plates, eachgripping plate has a gripping surface with one or more nubs.
 27. Thesystem of claim 22, wherein the controller is arranged to process dataobtained by the sensor, to calculate respective 3D position of the oneor more markers.
 28. The system of claim 22, wherein the controller isarranged to: compare data obtained by the sensor with reference data forthe one or more markers to determine similarity; binarizing thesimilarity result based on a threshold and applying a morphologicaloperation to the binarized result; and determine presence of the one ormore markers.
 29. The system of claim 22, wherein the controller isarranged to: compare data obtained by the sensor at different instancesusing a classifier, wherein the classifier is trained using dataobtained by the sensor at different instances.
 30. The system of claim21, wherein the deformation model is determined based on online learningand the mapping is based on a Gaussian mapping function.